An Analysis of the Memory Mechanism of XCSM

We analyze the memory mechanism of XCSM, the extension of XCS with internal memory. Our aim is to explain some of the results reported in the literature, which show that XCSM fails to learn an optimal policy in complex partially observable environments. The analysis we present reveals that the XCSM’s memory management strategy cannot guarantee the convergence to an optimal solution. We thus extend XCSM introducing a novel hierarchical exploration technique and modifying the technique used for updating internal memory. We apply the novel version of XCSM, called XCSMH, to a set of partially observable environments of different complexity. Our results show that XCSMH is able to learn an optimal policy in all the environments, outperforming XCSM in more difficult problems.